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This article describes the general approach that was taken in 
integrating and system testing the 3B20D Processor system. Since 
both the system hardware and software were developed simultane- 
ously, the goals of the system test and integration plan naturally 
shifted emphasis and expanded their scope from achieving hardware 
stability to establishing software functionality and finally to demon- 
strating system stability. This article also overviews some of the 
project management techniques and procedures applied during the 
development of the 3B20D Processor. 

I. INTRODUCTION 

An important aspect of the development of any complex system 
such as the 3B20D Processor is the methodical integration and system 
testing during all phases of the development consistent with the 
experience gained from previous developments. 1-4 Since the hardware, 
software, and microcode were designed and developed simultaneously, 
the initial efforts focused primarily on the hardware and firmware 
using stand-alone exercise modules and system diagnostics run from a 
laboratory support processor. After the hardware reached sufficient 
stability, emphasis turned to functional testing of each major software 
subsystem and feature. Finally, as full functionality was achieved, the 
major thrust of testing focused on system integrity and reliability using 
the previously developed tests as a regression test package to assure 
no loss in functionality as problems were cleared. The development 
methodology is summarized in the relative timeline sequence chart 
shown in Fig. 1. 

Also discussed in this article are some of the project management 



399 



DEVELOPMENT 



INTEGRATION 
HARDWARE 



TEST 



FACTORY 
SYSTEM TEST 



SYSTEM 
REQUIREMENTS 



j^JVCTIVITY* _J J_ 



— 4 t- J 



SYSTEM 
ANALYSIS 



RELEASE 



DEVELOPMENT CERTIFICATION 'INTEGRATION 1 SYSTEM TEST 
SOFTWARE 

Fig. 1 — Generalized development model. 



techniques and administration tools used to control the changes and 
new features introduced into the system. 

II. EARLY HARDWARE/SOFTWARE INTEGRATION AND TEST 
STRATEGY 

2.1 Objectives 

The objective of the initial integration and test effort on the proto- 
type hardware was to verify basic instruction execution and memory 
access, establish full diagnostic capability of the hardware, 5 prove in 
peripheral access and functionality, and establish stable communica- 
tion interfaces. In achieving these objectives, a stable software devel- 
opment environment was achieved for the major portion of the soft- 
ware development. 

2.2 Stand-alone exercise modules 

The diagnostics were developed to initially run from the laboratory 
support processor in conjunction with the hardware development. This 
simultaneous development of the diagnostics and the processor hard- 
ware had the unique advantage of providing individual functional 
verification of each circuit pack or major unit before integration of the 
operational system was attempted, thereby saving much laboratory 
time ferreting out faulty hardware. The initial functional integration 
started with simplistic CPU test modules that afforded stand-alone 
verification of basic operation. Upon reaching acceptable functionality, 
stand-alone test modules were used to establish communication with 
the disk file controller and moving head disks. 

2.2. 1 Central processing unit integration 

Two test modules were used extensively to integrate the early 
Central Processing Unit (CPU) hardware, firmware, and subsequent 
changes. The first test module was designed to test basic main-memory 
access and instruction execution with output to serial channel on the 
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Central Control Input/Output (CCIO) bus. 6 Loading this module from 
the laboratory support processor verified the communication link from 
the support processor to the 3B20D Processor. In addition, the exe- 
cution of the module not only verified basic hardware functionality 
but also verified the data-link capability to a TTY via the serial 
channel. The second test module, in combination with a primitive 
version of the operating system, established two processes and cycli- 
cally sent messages between them. This capability not only tested 
more of the hardware features of the CPU, but also provided a means 
to verify stable operation over long periods of time. This test module 
was then expanded to verify memory update on the off-line Control 
Unit (CU) and "soft switch" capability between the duplex units. 7 

2.2.2 File system integration 

Once basic operation of the CPU was verified, attention was pointed 
toward the file-system operation requiring integration of the Direct 
Memory Access (DMA) unit, the Disk File Controller (DFC) unit, and 
the Moving Head Disk (MHD). 8 Again a stand-alone test module, 
based on the disk driver software and the primitive operating system, 
was used for the integration of the hardware and firmware. Because of 
the large percentage of the hardware that had to be operational for 
successful execution of this test module, it became an invaluable tool 
not only for the integration of the preproduction hardware but also for 
Western Electric manufacturing, testing, and installation of early 
models of the 3B20D Processor in application system laboratories. 

2.3 System software 

Once the hardware was integrated and verified to the limits of the 
stand-alone test modules, development of the operating system and 
system-initialization software proceeded rapidly, and the integration 
effort switched emphasis from strictly hardware to system software. 
The strategy was to incrementally integrate — from the primitive op- 
erating system — each new capability of the operating system and 
system-initialization software with the hardware until a fully cycling 
stand-alone basic processor system was achieved. 

With the basic capability to initialize the system and cycle the 
operating system, integration proceeded to verify the 3B20D resident 
diagnostic control structure and diagnostics. 

By this time, additional integration tests were necessary to more 
fully expand coverage of the system. Thus, a test process was developed 
that created disk read and write jobs with a variable number of these 
child processes specifiable up to the number of allowable Dispatch 
Control Table (DCT) entries. Because of the large percentage of the 
processor used by this test process and because of the controllable 
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activity, it became an invaluable regression-test vehicle for subsequent 
integration activities as well as a system stress test. 

2.4 Results 

The primary result of this early effort was the establishment of a 
stable hardware and operating software base for the development of 
the features. 

III. INTEGRATION AND SYSTEM TEST 

The 3B20D system-level testing is actually divided into three distinct 
functional groups consisting of System Integration, System Test, and 
System Analysis. A brief historical review of the evolution of these 
groups is perhaps the best way to describe their respective functions. 
In early 1979 a decision was made to delegate the system testing of 
DMERT to Western Electric. 9 A Western Electric department was 
formed with the goal of taking over full responsibility for DMERT 
system testing by January 1, 1980. This transition actually took place 
about six months ahead of schedule in July 1979 and the system 
remains a Western Electric responsibility. The goals of the system 
testing group at that time were to release laboratory quality prereleases 
to DMERT applications to allow parallel application software devel- 
opment with the DMERT development. The system testing group 
also developed an extensive, documented set of tests that could be 
used not only to test the prereleases but would also serve as a base for 
testing all generic software releases in the years to follow. 

Another group, the System Integration group, was responsible for 
planning and coordinating the building (compiling) of each DMERT 
release, getting the release installed and cycling in the 3B20D devel- 
opment labs, and assuring that basic functions worked. Once this was 
accomplished, responsibility for the detailed testing was turned over 
to the system testing group. Thus, the system testing group could 
concentrate more on actual testing and problem resolution and less on 
bringing up the internal loads. 

3.1 Integration 

System integration controls the flow of software changes from the 
time a developer completes a change through the release of that 
change to a customer. The specific areas of responsibility include: 

(i) Load engineering and planning 
(ii) Benchmark tracking and analysis 
(Hi) Integration testing 
(iu) Release-letter generation 

(u) Modification Request (MR) tracking and MR data base integ- 
rity. 
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3.1.1 Load engineering and load planning 

For each DMERT release, an individual is assigned to be the load 
engineer. This individual serves as the focal point for all load-building 
activities. Specifically, the load engineer analyzes all changes planned 
for the release by the generic engineer, decides in what sequence 
changes should be taken, oversees the building of the load, and 
coordinates installation and integration testing of the load in the 
development labs. 

Members of the integration team report to the load engineer who 
assures that all activities needed to deliver the load on schedule are 
assigned and completed. The load engineer with assistance from the 
integration team resolves daily problems and, as necessary, reschedules 
activities and people. 

The load engineer in conjunction with the program administration 
staff coordinates the actual building of the load. The load engineer 
must thoroughly understand the mechanics of how the system is built, 
what software dependencies exist and how source code is controlled 
via the CMS/M2 system. 10 

3.1.2 Benchmark tracking and analysis 

Each new generic feature or major software enhancement results in 
a set of benchmarks that identify the date at which major activities 
are scheduled for completion. Benchmarks serve a dual purpose: first, 
as a management tool for measuring how the project is doing relative 
to the plan; and second, as a planning aide for other people or groups 
identifying dependencies for other features, hardware availability, or 
lab installation. 

Several tools have been used for identifying, tracking and reporting 
on feature benchmarks within the DMERT development organization. 

3.1.3 Integration testing 

One of the major objectives of the integration team is to assure that 
the load given to the system testing group is of sufficient quality to 
allow detailed functional testing. To verify that the system is of such 
quality, basic functional tests are run to assure that the major subsys- 
tems are operational. These include diagnostics, processor duplex 
operation, disk and I/O capabilities, and Recent Change and Verify. 

3.1.4 Release-letter generation 

Typically the applications that use the 3B20D Processor want the 
new DMERT software releases as soon as possible after the completion 
of system testing. This has presented a unique challenge to DMERT 
development management: the need to get releases, complete with 
essential documentation, to a number of different customers within 
one day of the completion of system test. 

SYSTEM TEST 403 



One vehicle used to supply necessary timely documentation to the 
customers is the release letter. This letter has evolved into a rather 
detailed document covering: 

(i) Support processor installation procedures 

(ii) 3B20D installation procedures 

(Hi) List of all file names 

( iv) List of all changed files 

(v) List of all required data base changes 

(ui) MR descriptions for all MRs resolved in the release 

(vii) MR exceptions list. 

Of particular importance is the MR exceptions list. The intent of 
this list is to communicate to the customers known problems that exist 
in the release and, when available, action to be taken if it is observed 
on their machines. This communication mechanism saves many hours 
that applications personnel would spend analyzing problems already 
identified by the DMERT organization. 

To assure timely distribution of this letter, all sections are put on a 
support computer and support programs are executed to assemble 
them into a document that is available on the day of the software 
release. 

3. 1.5 Detailed MR tracking and data base integrity 

The integration team also was chartered to establish the integrity of 
MR data base, to produce accurate and timely reports, and to respond 
promptly to high-priority problems. Weekly audits of the entire data 
base are performed to assure that MRs do not remain in a transient 
state for an unreasonable length of time. 

3.2 System test 

The primary objective of the 3B20D System Test group is to test 
the DMERT system on the 3B20D Processor in order to validate that 
all advertised features and capabilities perform according to their 
documented requirements. System tests are designed to test all the 
functional capabilities of the processor and its hardware both in no- 
load and stress environments. 

In the two and one half years since its inception, the System Test 
group has developed a complete system testing package containing 
over 700 test cases. As new features are developed, test cases are 
developed and each feature is thoroughly tested. Test cases are docu- 
mented and in many cases processes are written to automatically 
execute the tests. Once a feature is released for customer use, a subset 
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of the defined test cases is included as part of an on-going regression- 
test package. 

A concept of certification testing was established to identify prob- 
lems early in the development cycle. This allowed more problems to 
be debugged and fixed before release and resulted in a more stable 
system testing environment and higher ultimate product quality. 

Certification testing requires the developers to build in the official 
environment 10 and to demonstrate the proper execution of their new 
code to a system tester before it can be delivered to the integration 
team. The system tester has an option to request particular tests to be 
run with the new code and thus certify that the software to be 
submitted has passed some basic tests and can be approved for further 
processing. Software not passing certification is rejected and the de- 
velopers have to correct the deficiencies and schedule a follow-up 
certification test. 

3.3 System analysis effort 

The System Analysis Group (SAG) effort was planned as an exten- 
sion to Integration and System Testing. As its objectives, SAG was to 
perform tests aimed at measuring the performance and reliability of 
the 3B20D as a system. A separate development laboratory was 
constructed with the primary intention of simulating and functioning 
as a field site. Since this was the only 3B20D laboratory planned to 
run for long periods of time without rebooting, many problems of a 
periodic or long-term nature were first observed there. 

SAG members approached the stability aspect of the job by first 
defining measurable metrics. Objectives were defined based on the 
measured system reliability. The SAG team then identified and inves- 
tigated problems that impacted system reliability and reported the 
effects on system stability once the problems were resolved. 

Stability data was collected during weekend testing. The tests in- 
volved running a controlled-load package containing system exercise 
processes for specified periods of time, usually several days. These 
tests were generally run unattended to evaluate hands-off machine 
performance. All messages to the Read Only Printer (ROP) were 
stored on disk, dumped at the end of the test and analyzed using a 
program developed for this purpose. 

Three sets of objectives were defined for data analysis: a long-term 
objective for system reliability; a cut objective that identified satisfac- 
tory stability levels for first application at in-service offices; and the 
objective of establishing concern thresholds. Any data above the 
concern threshold was clearly unacceptable for even initial in-service 
machines. Data lying in the area between the cut objective and the 
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concern thresholds needed additional understanding in order to make 
a go/no-go decision on cutover. 

An example of one of the metrics used to track stability is shown in 
Fig. 2 for ten releases of DMERT prior to the first machine cutover in 
September 1981. 

IV. FACTORY SYSTEM TEST 

Factory System Tests (FST) and Quality Assurance (QA) tests are 
the final hardware tests run at the Western Electric Company manu- 
facturing plants to assure that a quality hardware product is delivered 
to the customer. 

4.1 Objectives 

The objectives of FST and QA are to test the hardware functionality 
and interconnections of fully assembled systems to assure that the 
processors as built meet design intent. These extensive tests assure 
the highest possible quality in the product when shipped to the 
customer. 

4.2 FST test strategy 

Instead of developing special test software for the FST, the actual 
DMERT operating system is enhanced with additional exercise proc- 
esses to form the Factory/Installation Software Test (FIST) package. 
The testing is divided into two phases: the normal operation phase 
and the stressed operation phase. These tests apply to all hardware 
delivered by the factory including the system as ordered, the comple- 
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Fig. 2— Interrupt incidence history. 
406 THE BELL SYSTEM TECHNICAL JOURNAL, JANUARY 1 983 



ment of spare circuit packs, growth units and circuit packs, and 
repaired product. 



4.2. 1 Normal operational tests 

The normal operational tests are designed to verify the functionality, 
interconnections, and basic maintenance operations associated with 
each unit under normal operating conditions. Included in these tests 
are the activation of system initializations under all possible minimum 
configurations using the power switch and the craft interface terminal. 
The tests then assure functionality of all units under simulated main- 
tenance conditions by removing and restoring each unit using both the 
power switch and the craft-interface terminal. During this test the 
system must remain operational. The next phase of testing requires 
the running and passing of all diagnostics for each unit within the 
system. Finally a series of special exercise processes are used to 
simulate actual operation of the disks, tape units, TTY and other data 
link controllers, and a CU soft-switch process for duplex capability 
verification. 



4.2.2 Stressed environmental operational tests 

The 3B20D Processor is designed to operate under a wide range of 
temperature and battery conditions. To assure that the system meets 
the design intent to operate under these conditions, two additional test 
environments are imposed on the machine before shipment. 

4.2.2.1 Low voltage. The power converters are stressed most under 
conditions of low-input voltage; thus, the system must pass all the 
tests prescribed above at an input voltage of -43.75 ± 0.05 volts. This 
voltage is 91 percent of the nominal —48 volts. 

4.2.2.2 High temperature. High-temperature operation of the 3B20D 
Processor is critical to avoid outages during commercial power or 
mechanical failures that result in the loss of building air-conditioning 
systems. The system tests prescribed above must pass in a system that 
has been operating at a stable elevated temperature of 49 °C ± 1°C for 
a period of at least four hours. 

4.3 QA testing 

In addition to the factory system test on all systems, additional tests 
are rerun under the auspices of the Bell Laboratories quality assurance 
organization and the Western Electric quality review organization to 
assure that statistical quality control limits are not exceeded, thus 
maintaining a high level of quality for the customers. 
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4.4 Result 

A major milestone was achieved in March 1980, when the first field 
shipment to the Traffic Service Position System (TSPS) site in San 
Antonio, Texas, was not only delivered on schedule, but passed the 
complete battery of factory system tests. 

V. ADMINISTRATION 

In this section, a brief overview of some of the important aspects of 
project-management and project-control techniques are presented. 

5. 1 Change authorization 

From the beginning of the project, the hardware design was under 
very tight controls. All changes or feature additions had to be approved 
by a management-change committee with representation from Bell 
Laboratories and Western Electric. This committee provided both a 
forum to review designs and design changes and to discern the eco- 
nomic impact of each change. This committee then established a joint 
subcommittee, called the Engineering Support Group, to schedule and 
track each change from design through manufacture and ultimately to 
the installation into the various system development laboratories. 

Software change control was less tightly controlled during the initial 
development and relied heavily on the software development super- 
visors responsible for each subsystem. Once the software was delivered 
to the application more stringent controls were introduced. At that 
point, feature content, overall coordination, and generic scheduling are 
the responsibility of the Generic Engineer and the Project Manager. 

5.2 Application interlaces 

To assure that the 3B20D Processor system meets the needs of the 
variety of Bell System applications, a group was established to act as 
the single focal point for the applications for all feature requests and 
MRs. 

5.2. 1 Feature content 

To establish feature content of the system, the Application Interface 
group, in concert with the applications, developed a prioritized list of 
feature requests and enhancements for the Project Manager and the 
Generic Engineer to review. Thus, a final list of features and enhance- 
ments was established taking into account customer needs, schedules, 
and resource limitations. 

5.2.2 Modification requests 

Initially the Application Interface group also acted as a clearing- 
house to prioritize, from the users point of view, the problems that 
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they discovered as the generic matured. This list, in conjunction with 
internally generated MRs, formed the basis for the Generic Engineer 
to approve MRs to be fixed for inclusion in the generic. Once an MR 
was approved, the Load Engineer tracked its progress through devel- 
opment, integration, system test, and release. 

Once the first generic was cut into service, a committee was estab- 
lished with representation from applications, DMERT development, 
generic engineering, system test, load engineering, and field support. 
This committee's function was to tightly control and adjudicate all 
software changes so as to assure that field service was not adversely 
affected and that real service problems were quickly attended to and 
delivered on a timely basis. 

5.3 Project-tracking tools 

A finite-state MR control mechanism was put into place to track 
and record changes in the status of MRs during the development 
cycle. 10 From this data base, various reports were automatically gen- 
erated for use by all organizations associated with the project. This 
central source of project-status information was an essential ingredient 
to the determination of areas of concern so that action could be taken, 
as well as a repository of all schedule information relating to MRs. 
This capability formed the nucleus of the automated project-manage- 
ment tools. 



VI. CONCLUSION 

The 3B20D Processor is operating effectively in the field since the 
first cutover in September 1981. The rapid field buildup during the 
first six months (24 machines cut into service) could not have been 
possible if all parts of the system were not of the highest quality and 
designed for high reliability. Much of the success of the project is 
attributed to the extensive testing both by the DMERT development 
organization, Western Electric organizations and application organi- 
zations during each step of the system's introduction. 
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